High-speed imaging with a sub-sampled array of image sensors

ABSTRACT

A method to capture microscopy images from multiple image sensors and to relay them to one or more central processing units with high frame rates can include preprocessing the image data from the image sensors to reduce the sizes of the captured images, before sending the image data to a central processing station for analysis. Using multiple image sensors and an image reduction process, large image frames of over 20 megapixels and up to 1 gigapixel or more can be obtained at high imaging frame rates of 30 or more to up to hundreds or thousands of frames per second.

The present patent application claims priority from U.S. ProvisionalPatent Applicant Ser. No. 62/965,890, filed on Jan. 25, 2020, entitled“High-speed imaging with a sub-sampled array of image sensors”, of thesame inventors, hereby incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

Most digital imaging technologies to date rely on a single image sensor,which includes an array of light-sensitive pixels, such as CMOS or CCD.Current sensors typically include millions to several tens of millionsof pixels, to form images with several to tens of megapixels ofinformation. While the cost of such multi-megapixel CMOS image sensorshas dramatically decreased over the past decade, from hundreds ofdollars to fractions of a dollar per sensor, the number of availablepixels per sensor has not dramatically increased. That is, it is stillnot possible to easily obtain image sensors with multiple hundreds ofmegapixels. While several sensors with up to 100-200 megapixels exist,due to manufacturing difficulties and low yields, these sensors remainprohibitively expensive (tens of thousands of dollars) for manyapplications.

At the same time, all current image sensors of any size exhibit alimited frame rate (i.e., the number of frames containing a set of Ndigitized pixel values from the sensor array per second). This limitedframe rate is caused by a number of limitations, including a limitedrate of information that can be read off of the sensor per second, alimited data transfer rate from off of the sensor to another location,and potential limits faced by heating or other effects, for example. Ageneral survey of currently available sensors suggests that the productof the maximum frame rate of a sensor M and the number of pixels withineach frame N faces a practical upper bound D of approximately 1Gigabyte/second (that is, MN<1 GB=D). There are certainly exceptions tothis limit, but it is in general observed across many currentlyavailable CMOS sensors.

Thus, there is a need for a technology that can circumvent thislimitation to offer very large frames (N>10 MP or more) at high framerates (>30 fps, and up to thousands of fps).

SUMMARY OF THE EMBODIMENTS

This patent proposes a system and method to obtain video images at aframe rate that is faster than standard video rates (30 or more framesper second), but with a very large number of image pixels per frame(over 20 megapixels, and up to 1 gigapixel or more). The proposedinvention is comprised of more than one digital image sensor, each withan associated lens that images a unique field-of-view (i.e., a uniquearea within object space), which routes data to a centralized processinglocation that can both control the sensor properties, and direct imagedata collected by each image sensor to computer memory. Collected imagedata from each sensor can be tiled together to form a full image of theobject of interest. Each sensor can be configured to return image dataat a high frame rate by reading from a subset of its pixels. Thecombination of these two insights yields a system than can captureimages, for example, with tens of megapixels at hundreds to thousands offrames per second.

In some embodiments, the present invention discloses methods and systemsto capture microscopy images from multiple image sensors and relay themto one or more central processing units with high frame rates. Themethod involves preprocessing the image data from the image sensors toreduce the sizes of the captured images, before sending to a centralprocessing station for analysis. Using multiple image sensors and animage reduction process, large image frames of over 20 megapixels and upto 1 gigapixel and high frame rates of 30 or more to up to hundreds orthousands of frames per second can be obtained.

In some embodiments, the methods can offer a system, such as microscopesystem, with the ability to select desired final image pixel count anddesired frame rate for a large frame captured by multiple cameras. Themicroscope system can also include fast data transfer, which includesparallel data transfer from the cameras, organizing data into datapackets having the size of a partial image frame, selecting datatransfer interfaces for matching bandwidths, and direct memory accessfor sending image data directly to the memory of the central processingstation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a schematic configuration for a system according tosome embodiments.

FIGS. 2A-2C illustrate flow charts for a system with large frames andhigh frame rates according to some embodiments.

FIG. 3 illustrates a data flow for an image reduction process accordingto some embodiments.

FIG. 4 illustrates a sub sampling process for an image reduction processaccording to some embodiments.

FIG. 5 illustrates a windowing sub sampling process for an imagereduction process according to some embodiments.

FIG. 6 illustrates a sub sampling process for an image compressionprocess according to some embodiments.

FIGS. 7A-7B illustrate flow charts for a system with large frames andhigh frame rates according to some embodiments.

FIG. 8 illustrates a schematic microscope system according to someembodiments.

FIGS. 9A-9B illustrate a configuration for a microscope system accordingto some embodiments.

FIG. 10 illustrates a flow chart for operating a microscope systemaccording to some embodiments.

FIG. 11 illustrates an operation schematic of a microscope systemaccording to some embodiments.

FIGS. 12A-12C illustrate a sequence of data flow from a camera arrayaccording to some embodiments.

FIG. 13 illustrates a microscope system configured with an imagereduction capability according to some embodiments.

FIGS. 14A-14B illustrate flow charts for forming microscope systemsaccording to some embodiments.

FIG. 15 illustrates a correlation between image reduction and imageframe rates according to some embodiments.

FIG. 16 illustrates a correlation between image reduction and imagequality with image frame rates according to some embodiments.

FIGS. 17A-17B illustrate flow charts for optimizing image parametersaccording to some embodiments.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In some embodiments, the present invention discloses systems and methodsto capture images having very large frames, such as greater than 10 MP(megapixel) or more, such as up to gigapixels, and at high frame rates,such as greater than 30 fps, and up to thousands of fps. The imagecapture process can also provide an image reconstruction of a sampleusing a spatial-angular distribution of light reaching the sample tooffer at least a measure of sample depth, spectral (e.g., color)properties, optical phase at the sample plane, or a higher imageresolution than the that defined by the diffraction limit. Further, theimage capture process can offer an optimization between large frames,high frame rates, and image reconstructions.

The image capture process can achieve large frames, high frame rates,and image reconstructions through an optimized data transfer between theimage sensors, which are configured to capture the images, and acomputational system, which is configured to process the image data.

In some embodiments, the image capture process can be used in a systemhaving more than one digital image sensor, each with an associated lensthat images a unique field-of-view (i.e., a unique area within an objectspace), which routes data to a centralized processing station. Thecentral processing station can control the image sensor properties, suchas providing imaging parameters to the image sensors, and direct imagedata captured by each image sensor to a memory of the central processingstation. The captured image data from the image sensors can be tiledtogether, such as using a stitching process which aligns features fromthe spatially overlapping portions of the image data, to form a fullimage of the object of interest. The full image can have a very largeframe size or can image a large area of the sample, through the tilingof multiple image sensors. The system can be operated in a videorecording mode, wherein multiple full images are acquired over time. Theuse of multiple image sensors can require a high data transfer ratebetween the image sensors and the central processing station, inparticular when operated in video recording mode, which can be achievedthrough a careful design of the system hardware and software.

The image sensor can be configured to return image data at a high framerate, such as at a desired frame rate determined by a user inputted tothe central processing station, by using an image reduction process,such as a sub sampling process or a binning process, for the capturedimage data, such as reading from a subset of its pixels. The use of animage reduction process can reduce the requirements of the high datatransfer rate per each set of image frames captured by the multipleimage sensors, which can allow the system to have a high video framerate. The combination of using multiple image sensors and an imagereduction process can yield a system than can capture images with up totens to hundreds of megapixels at hundreds to thousands of frames persecond.

The image sensor can be configured to capture images that are producedunder illumination from unique spatial-angular distributions of light,e.g., through an illumination source configured to shine light on thesample from multiple angles and spatial patterns, so that the spatialand angular properties of light reaching the sample changes over time.The images captured under the unique spatial and angular distributionsof illumination can be processed to reconstruct an image of the sample,with improved image characteristics. The use of multiple variablyillumination patterns can also require a high data transfer rate betweenthe image sensors and the central processing station, which can beachieved through an optimization of image acquisition hardware, dataaggregation digital logic, firmware, and integration with data postprocessing on the central processing station.

In some embodiments, the use of an image reduction process with multipleimage sensors and variably illumination patterns can offer a datatransfer rate suitable for capturing images having a large image frame,high frame rates and improved image characteristics. Further, the systemcan offer optimized frame rates and image characteristics through acompromise between the sub sampling rate of the sub sampling process andthe number of illumination patterns of the variably illuminationpatterns.

In some embodiments, the image reduction process can be a way topotentially speed-up the process of patterned illumination. For example,a system using 54 unique 13 megapixel image sensors to achieve a largeimage frame can capture and save 6 full frames per second. To improveimage characteristics, such as to obtain depth measurements or higherresolution, a variably illumination setting can require 24 differentillumination patterns, which would take 4 seconds for capturing fullframe image data for an image reconstruction of 24 image frames, eachcaptured through a different illumination pattern. The 4 second time canbe too long for some fast-moving organisms, since there can be smallmovements within that time window. The image reduction process canassist in speeding up the data acquisition process to allow the analysisof the fast-moving organisms. For example, with a sub-sampling processhaving a sub sampling factor of 16 (e.g., 4×4 sub-sampling, which willreduce the total number of saved pixels within each image frame by 16),the total image acquisition time can be reduced by a factor of 16, e.g.,to 0.25 seconds to capture 24 uniquely illuminated image frames, whichcan be acceptable for many applications.

General Concept

In some embodiments, the present invention discloses systems and methodsof capturing large image frames with high frame rate in an array ofmultiple small microscopes (i.e., micro-cameras), tiled together in anarray. Using a tightly packed array of micro-cameras, high resolution(1-10 μm) over a large area (hundreds of square centimeters) can beachieved. Using an image reduction process including a sub samplingprocess, a binning process, or a data compression process, high framerates of up to hundreds or thousands of frames per second can beachieved.

FIG. 1 illustrates a schematic configuration for a system according tosome embodiments. The system 100, such as a microscope system, caninclude an array of cameras 110 and a patterned illumination source 121and 122 for microscopic imaging. The cameras 110 are configured to imagepartially overlapping field of views (FOV) 111 of image areas of asample 120. With multiple cameras, and for a fixed resolution setting,the system can have multiple times larger field of view, as compared tosystems having a single camera. For example, for a microscope systemhaving n cameras, the captured images can have an approximately k timeslarger field-of-view per snapshot than a standard microscope.

In some embodiments, the system can be configured to have the camerassending out image data in parallel, to thus achieve a faster datatransfer rate as compared to a single camera having a large imagesensor. For example, the microscope system can capture and process 800MP per frame at 5 frames per second due to the parallel image datatransfer from the multiple cameras.

In some embodiments, the cameras can be configured for pre-processingthe capture image data, e.g., processing the data before sending thedata out to a central processing station for analysis. For example, thecameras can include a pre-process module, which can configured to sendout the full image data 150, to send out reduced image data 160, e.g.,sending out pixels 161 and skipping pixels 162, or to send outcompressed image data 170, with the compressed data 171 having a smallernumber of bits than the full image data 150, but carrying essentiallythe same amount of information (e.g., lossless compression or a losslessJPEG compression) or carrying the essential information (e.g., lossycompression or lossy JPEG compression) of the full image data.Alternatively, compressed image data may simply be a windowing of thesensor imaging area, for example from a sensor that originally includesM×N pixels, only pixel data from the center M/2×N/2 area of the sensormay be read off to decrease the total amount of image information.

In some embodiments, the image reduction process can include a subsampling process, in which an image is reduced in its dimensions toobtain a smaller image. The sub sampling process reduces the image sizeby removing information all together, e.g., removing the subsequentpixels in both directions. For example, a 2×2 sub sampling process cankeep one pixel for every 4-pixel area, resulting in a factor of 4 inimage size reduction.

In a sub sampling process, the pixels in an image can be thrown away insubsequent or sets of repeating rows or columns to create a reduced sizeimage. For example, to create a half-sized image, e.g., an image havingsizes in both x and y dimensions being half the original sizes, the subsampling process can throw away every other row and column, forming a2×2 sub sampling process. When the sampling rate gets too low, e.g., thesub sampling image becomes too small as compared to the original image,the details in the original image cannot be captured in the sub samplingimage anymore. Thus, the sub sampling process can have limits to stillproduce a full-fidelity imaging result, which is called the Nyquistrate, which specifies the minimum signal or image rate.

In the sub sampling process, an interpolation or a smoothing process canbe used to reduce aliasing. When the original image has highfrequencies, e.g., high details, and the pixel sampling rate is not highenough to capture the details, the sub sampling process can generateincorrect signals or images, e.g., an alias. Thus, the sampling rateshould be high enough to capture the highest frequency in the image. Toavoid aliasing, the sampling rate can be greater or equal to the maximumfrequency in the image, or can be equal or greater than 2× the maximumfrequency. Alternatively, the image can be subjected to a filterprocess, such as a low pass filter or a Gaussian filter to blur theimage, before being sub sampled.

The image reduction can include an image binning process, in which theimage size is reduced by averaging the information in each bin ofpixels, such as forming a super pixel for each bin. For example, theimage can be partitioned into non-overlapping M×N tiles, where M and Nare the numbers of rows and columns of a tile, and each tile getsreplaced with a representative value, such as the average values of thepixels in the tile. The binning process can reduce minor fluctuationscaused by noise because of the averaging process.

Thus, with an image reduction process, the image frame rates of thesystem can be significantly increased. For example, the maximum framerate vs binning or sub sampling curve is approximately linear but withsmall deviations. Thus, a microscope system capable of processing 800 MPper frame at 5 frames per second, using a binning process of 2×2 forV=4, can offer 200 MP images at 20 frames per second, which is nearlythe video rate of 24 frames per second. The high frame rates at highpixel counts can open up new applications for example, in observing fastmechanisms of living organisms.

The sub sampling process can be based on an image perception, e.g., byreducing the resolution for features that are not highly noticeable, theimage sub sampling process can reduce the image size with lessnoticeable loss of image quality. For example, in color images encodedwith luminance and chrominance, the chrominance can be sub sampled toreduce the image size since changes in chrominance are less noticeablethan luminance. For example, the chrominance values can be sub sampledby ½ or even ¼ of that of the luminance, e.g., the intensity. Even withsuch as a rather high sub sampling rate, the differences in terms ofperceived image quality are hardly noticeable. Moreover, in color imagesensors with a Bayer filter pattern, sub-sampling by pixel binningacross ach 2×2 Bayer filter pattern area can produce a grayscale imagewith 4× fewer total pixels than what is required to read off the sensorfor a color image, while yielding a 4× increase in imaging frame ratefor the grayscale image.

The image reduction can include an image compression process, which canreduce the size of the image in a loss or lossless compression. Theimage compression process can reduce the image size, such as reducingthe quality of the image by discarding unnecessary data, such a limitingthe colors used in the image.

In some embodiments, the image reduction process, e.g., the sub samplingprocess, the binning process or the image compression process, can beperformed in a pre-process module in the camera, such as integrated withthe camera image sensor, or closely interfaced with the image sensor, toprevent delays in data transfer from the image sensor to the centralprocessing station. For example, the sub sampling process can beperformed by a module coupled to the image sensors to select digitallight intensity value from a pixel and skip other pixels, such asskipping every other row and column in a 2×2 sub sampling process. Thebinning process can be performed by combining digital light intensityvalues from adjacent pixels in the image sensor, such as in a CCD(charge coupled device) or CMOS device, to reduce the total number ofbits per frame that must be read off of each sensor device. The lowpass, Gaussian filter, or the data compression can be performed withappropriate logic devices. For example, a field programmable gate array(FPGA) can be integrated with the image sensor to perform imagecompression. The integration of the image reduction process with theimage sensor can significantly reduce delays in performing the imagereduction process.

FIGS. 2A-2C illustrate flow charts for a system with large frames andhigh frame rates according to some embodiments. In FIG. 2A, operation200 forms a computational microscope, with the microscope including acamera array having multiple cameras illuminated by one or morepatterned illuminations from an illumination source. The camera array isconfigured to send the images in full image frames or by reduced sizes,wherein the microscope further comprises a processor configured toprocess images captured by the camera array. The images can have thesizes reduced in a pre-processed module integrated with the imagesensors of the cameras in the camera array.

In FIG. 2B, operation 220 obtains a higher frame rate for large capturedimage frames by configuring a camera array to send reduced image sizes.The images can have the sizes reduced in a pre-processed moduleintegrated with the image sensors of the cameras in the camera array.

In FIG. 2C, operation 240 generates an illumination pattern on a sample.Operation 250 captures n images from n cameras, wherein at least twoimages of the n images are overlapped. Operation 250 sends full rawimages, portions of the full raw images, or processed images to acomputational unit for analysis. The images are sent in multipleparallel data streams to improve a data transfer rate. Operation 260optionally repeats for other illumination patterns.

General Data Flow for Sub Sampling

FIG. 3 illustrates a data flow for an image reduction process accordingto some embodiments. Multiple cameras 310 can be configured to captureimages 350, for example, through image sensors 312. The captured imagedata can be pre-processed, for example, by a pre-process module 313, togenerate full size images 350, or reduced size images, such as subsampling or binning images 360 or compressing images 370.

In some embodiments, the cameras 310 can receive instructions 334, suchas receiving image parameters, from a process module 330. Theinstructions can include instructions for sending full images or forsending reduced images, such as for sub sampling rate, binning rate,compressed rate, or any combination. The process module 330 can beconfigured to receive multiple data streams from the cameras inparallel, to have a high data transfer rate. The process module 330 caninclude multiple circuits configured to process the multiple datastreams in parallel.

In some embodiments, the pre-processing process can occur in modulesintegrated with the image sensors to minimize delays in data transfer.For example, the captured image data can be pre-processed to skippingpixels in a periodic or non-periodic manner. The captured image data canbe pre-processed to read a continuous subset of pixels in an area of thefull image frame. The captured image data can be pre-processed tocombine pixel intensities in each bin of pixels to form an averageintensity for the bin. The captured image data can be pre-processed tocompress the image data, or via another image data reduction process.

For example, the pre-processing process can be configured to removeadjacent or nearby pixels that do not have significant variation inintensities, e.g., pixels having similar intensities. The pre-processingprocess can be configured to remove adjacent or nearby pixels that havesmall variations in intensities. The removal process can be performed byfirst obtaining the spatial derivative of each image sensor frame, e.g.,calculating the intensity gradients between adjacent or nearby pixels.The pixels with the spatial derivative, e.g., intensity gradient, lessthan a pre-defined threshold value are then removed, saving only thepixel values and the pixel locations with a value greater than thepre-defined threshold value. In some embodiments, a pixel having arepresentative spatial derivative can be saved, to retain at least apixel with small intensity variations. The concept of spatial derivativecan be summarized by a 2D convolution kernel that takes the form of [−1,1; −1, 1], for a simple 2×2 convolution kernel, which is convolved withthe image to produce the spatial derivative output. This simple 2×2convolution kernel can be replaced by one or more convolution kernels ofdifferent sizes, e.g. taking the form of wavelet convolution kernels,for an alternative means to reduce image data.

In some embodiments, the removal process can be performed by firstobtaining the temporal derivative of each image sensor frame, e.g.,calculating the intensity gradients between pixels in sequential imageframes. The pixels with the temporal derivative, e.g., intensitygradient with respect to time, less than a pre-defined threshold valueare then removed, saving only the pixel values and pixel locations witha value greater than the pre-defined threshold value. Image frame datafrom a previous image capturing process by the same image sensor can bestored, such as in a buffer, to enable the temporal derivative removalprocess. The concept of temporal derivative can be summarized by a 1Dconvolution kernel that takes the form of [−1, 1], for a simple 1×2convolution kernel, which is convolved with the two image frames toproduce the temporal derivative output. This simple 1×2 convolutionkernel can be replaced by one or more convolution kernels of differentsizes with 2 or more frames, e.g. taking the form of wavelet convolutionkernels, for an alternative means to reduce image data In someembodiments, a pixel having a representative spatial derivative can besaved, to retain at least a pixel with small intensity variations.

In some embodiments, the image reduction process can include acombination of size reduction processes, such as a combination of imagesub-sampling, image binning, image removal of small intensityvariations, or image compression. The combination of size reductionprocesses, performed in a pre-processing module integrated to the imagesensors, can further reduce transmitted data rates and subsequently leadto higher total system frame rates.

FIG. 4 illustrates a sub sampling process for an image reduction processaccording to some embodiments. Multiple cameras 410 can be configured tocapture images 450. The captured image data can be pre-processed togenerate reduced size images, such as sub sampling or binning images460. The pixel sub sampling process can be used on each of theindividual image sensors to reduce the number of pixels that transmitoptical image data. For example, in a 3×2 sub sampling process, eachimage sensor can read off every third pixel in the first row of pixels(i.e., does not read pixels from columns 2 and 3, 5 and 6, etc.), andthen skips every other row (i.e., does not read pixels fromeven-numbered rows). Any number of rows and/or columns may be skipped.Further, the sub sampling process can use a pattern that is notperiodic. After sub-sampling by a factor of V (i.e., skipping V−1 pixelsfor every pixel read out), N/V pixels per frame are sent to the centralprocessing unit, with N being the number of pixels in the frame. Thisimplies that V more frames per second can be sent from each image sensorto the central processing unit within a fixed amount of time as comparedto imaging all N pixels per frame. For example, in the 3×2 sub samplingprocess described above, V=6, for an increase of 6 in frame rate.

In some embodiments, the image sensors in cameras 410 capture opticaldata, which is typically digitized on the image sensor, such as a CMOSsensor, and which is then sent as digital electronic signals in multipledata streams 435 to a process module 430, which can organize the data tosend to a memory 441 associated with a processor 440 of a centralprocessing station in a serial data stream 445.

In some embodiments, the process module 430 can be a field programmablegate array (FPGA), which can collect, re-arrange, and re-transmit theacquired image data to the memory in the serial data stream 445. TheFPGA can also send digital signals to the cameras, e.g., to thepre-process module in the cameras, to configure the image sensors tosub-sample from the image sensor array according to a configuration orat a sub-sampling rate, instructed by a user.

After the image data is saved in computer memory, a post-processingalgorithm may then, for example, combine the acquired micro-cameraimages together into a final composite image that has a larger FOV thanany of the individual micro-cameras individually. After post-processing,the final result can then be displayed to a user or stored in digitalmemory.

FIG. 5 illustrates a windowing sub sampling process for an imagereduction process according to some embodiments. Multiple cameras 510can be configured to capture images 550. The captured image data can bepre-processed to generate reduced size images, in which instead ofskipping pixels in a periodic or non-periodic manner, a continuoussubset of W pixels is read from the full image frame to obtain a window563, such as reading the center portion of the image from the imagesensors, to send to the process module. With the sub sampling processreads out image window 563 having W pixels, the frame rate is increasedby a factor of V=N/W, with N being the number of pixels in the frame.The cameras 510 can send multiple data streams to a process module 530,which can organize the data to send to a memory 541 associated with aprocessor 540 of a central processing station.

With a sub-sampling strategy, given a data rate limit D=NM, where N isthe number of pixels within each frame and M is the frame rate for eachsensor, and a selected sub sampling factor V for each sensor, then it ispossible to achieve a frame rate that is approximately MV from eachsensor. The increase in frame rate is at the expense of the reduction inpixels in the image frames, e.g., the frames from each sensor now onlyhave N/V pixels in them, for a reduction of V due to the sub samplingprocess.

In some embodiments, to increase the image size or a resolution, e.g.,to make up for the lost pixels created by the sub sampling process, atleast a factor of V additional micro-cameras can be added to the cameraarray. The additional micro-cameras can have the same field of view asthe original camera array, which results in a same image frame with theadded V factor of the pixels. The additional micro-cameras can haveoverlapped field of views, such as with the original camera array andwith each other, which results in a much larger image frame, e.g., Vfactor larger image frame.

After fusing the images from the camera array and the additionalmicro-cameras together, the system can create composite images thatcontain V*(N/V)=N pixels per frame, but now with a frame rate of MV,which can be much higher than the original frame rate, given V is large.

For example, a system can include k=54 micro-cameras in a camera array.For the image sensors having a number of pixels per sensor is N=20million, with the original frame rate of M=20 frames per second, using asub sampling factor of V=16, a faster frame rate VM=320 frames persecond can be achieved. For the high frame rate, the image framesinclude a reduced number of pixel per image of N/V=1.25 million. Withthe camera array of 54 micro-cameras, the final stitched composite imagecan contain up to k*(N/V)=67.5 million pixels, now with a higher framerate of VM=320 frames per second.

In some embodiments, the system can be configured to incur minor delayin data transfer flow with the additional micro-cameras, due to theparallel processing from the micro-cameras. For example, since themicro-cameras in the camera array are sending image frames to thecentral processing station in multiple parallel data streams, the dataflow rate can be similar for one or for multiple micro-cameras.

In some embodiments, an intermediate module can be disposed between themultiple cameras and the central processing station, for example, toorganize the image data in the multiple data streams and to function asan interface for the central processing station, due to the different IOconfigurations between a standard central processing station, such as astandard computer, and the micro-cameras.

The intermediate module can be configured to accept multiple datastreams 535 from the camera array, and can organize the image data, forexample, to form data packets to send to the memory of the centralprocessing station in a serial data packet stream 545. The intermediatemodule can include a chip set that can be configured to process multipledata streams in parallel, such as an FPGA, an ASIC, an ASSP, or a SoC.For example, an FPGA based module can offer flexibility in configuringto be the intermediate module.

With the FPGA based module capable of processing the multiple cameradata streams in parallel, the data transfer rate from the multiplecameras to the FPGA based module can be similar for one or for multiplecameras.

Further, the FPGA based module can be configured so that the bandwidth,or data transfer rate, of the serial data stream 545 is greater than orequal to the combined bandwidths of the multiple parallel data streams535, to avoid bottlenecking in data transfer flow from the cameras tothe memory 541. For example, the serial data stream 545 can include ahigh speed data transfer interface with the number of data lanesdesigned to exceed the data transfer rates of all parallel data streams535, such as PCIe-16 or PCIe-32. With matching bandwidths between thedata transfer from the multiple cameras to the FPGA based module, andfrom the FPGA based module to the memory of the central processingstation, the data transfer rate between the FPGA based module and thememory of the central processing station can be the same as the datatransfer rate between the multiple cameras and the FPGA based module,e.g., the data transfer rate between the multiple cameras and the memoryof the central processing station is not affected by the presence of theFPGA based module.

Further, the serial data stream can be configured to include small datapackets, with each packet having only a portion of an image frame, suchas a reduced image frame. The small data packets can enable rapid dataflow, with minimum delays in data accumulation at the process module.Also, the serial data stream can be configured to send the data packetsby direct memory access, e.g., without being processed by the processor530, to further reduce delays in the data flow due to the processor.

FIG. 6 illustrates a sub sampling process for an image compressionprocess according to some embodiments. In some embodiments, the imagereduction process can include sub-sampled pixels (or binning pixels)from each image sensor, as well as pre-process these sub-sampled imagepixels, to achieve an additional degree of data reduction. For example,multiple cameras 610 can be configured to capture images 650. Thecaptured image data can be pre-processed to generate reduced sizeimages, such as sub sampling or binning images 660. The images can bepre-processed to remove pixels having similar variation in intensities,or having small intensity variations, either on an image frame or on twoor more subsequent image frames. The images can be compressed to formcompressed images 670, either in the cameras or in the process module630. In some embodiments, the cameras 610 can send multiple data streamsto a process module 630, which can organize the data to send to a memory641 associated with a processor 640 of a central processing station.

For example, the pre-processing process can remove pixels that have lowvariations in intensities, either in space or time. The removal of lowintensity variations can be performed by spatial or temporal filteringvia convolutions and removal of pixel values below certain thresholds,for example by spatial or temporal derivatives of the pixel intensities,either within an image frame or in two or more consecutive image frames.In some embodiments, the pre-processing process can include imagecompression, such as a JPEG-type compression. The compression processcan be performed in a module integrated with the image sensor, or can beperformed in the FPGA base module.

FIGS. 7A-7B illustrate flow charts for a system with large frames andhigh frame rates according to some embodiments. In FIG. 7A, operation700 forms a computational microscope, with the microscope including acamera array having multiple cameras illuminated by one or morepatterned illuminations from an illumination source. A camera of thecamera array is configured to preprocess captured images to reduce sizesof the images before sending to a processor for analysis. The imagereduction preprocess can be performed in the camera, e.g., integratedwith the image sensor using hardware circuits to reduce delays in datatransfer. The image reduction preprocess can be performed in a processmodule, outside the camera, in a hardware component configured toparallel processing the multiple data streams from the cameras. The sizereduction can include at least one of a sub sampling process, a binningprocess, or a data compression process.

In FIG. 7B, operation 720 captures images by cameras of a camera array.Operation 730 processes the images to reduce sizes of the images, withthe size reduction including at least one of a sub sampling process, abinning process, or a data compression process. The image reductionpreprocess can be performed in the camera, e.g., integrated with theimage sensor using hardware circuits to reduce delays in data transfer.Operation 740 sends the processed images, in multiple parallel datastreams, to a process module for organizing into a data packet streambefore reaching a computational system for image analysis.

Micro-Camera Array Microscope (MCAM) System

In some embodiments, the present invention discloses a system havingparallel image data acquisition across an array of k separate imagesensors and associated lenses, which can route data to another computerlocation or to memory at much higher rates. For example, for a set ofsensors that each exhibit a maximum image data extraction rate D due tolimitations of on-chip data routing, the parallel data transferringprocess for k image sensors can allow for extraction of image data at arate kD.

In some embodiments, a parallel to serial module can be used to receiveand process the parallel data transfer from the multiple image sensors.For example, the parallel to serial module can organize the receivedimage data into a serial data packet stream, to send to a centralprocessing station, such as sending directly to the memory of thecentral processing station through direct memory access. The parallel toserial module can simplify the central processing station, for example,by allow the use of a standard computer or a standard data processingsystem without the need for a central processing system with multiplespecific interfaces for interfacing with the multiple image sensors.

At the same time, all current image sensors of any size exhibit alimited frame rate (i.e., the number of frames containing a set of Ndigitized pixel values from the sensor array per second). This limitedframe rate is caused by a number of limitations, including a limitedrate of information that can be read off of the sensor per second, alimited data transfer rate from off of the sensor to another location,and potential limits faced by heating or other effects, for example. Atthe present time, the currently available sensors suggests that theproduct of the maximum frame rate M of a sensor and the number of pixelsN within each frame faces a practical upper bound D of approximately 1Gigabyte/second (that is, MN<1 GB=D). There are exceptions to thislimit, but in general, this limit is observed across many currentlyavailable CMOS sensors.

In some embodiments, the present invention discloses systems and methodsto exceed this limitation to offer very large frames (N>10 MP or more)at high frame rates (>30 fps, and up to thousands of fps). The methodparallelizes image data acquisition across an array of k separatesensors and associated lenses, which can provide large image frames withminimum decrease in data transfer rate. Together with an image sizereduction for the captured image frames, the parallel array of k imagesensors can produce large image frames at high frame rates.

In some embodiments, a k-parallel-image sensor system can be configuredto obtain video images at a frame rate that is faster than standardvideo rates (30 or more frames per second), and with a very large numberof image pixels per frame (over 20 megapixels, and up to 1 gigapixel ormore). The system can use parallel image sensors to obtain large imageframes, and can use an image reduction process to increase the imageframe rate. Further, the system can be scalable and flexible, with theimage frame sizes and the image frame rates determined by the need of auser, for example, by selecting the number of image sensors suitable forthe desired image frame size, and by selecting the image reductionfactors suitable for the desired image frame rate. The image reductionfactor can also vary as a function of image sensor. For example, imagesensors located towards the center of the array of image sensors can usea low reduction factor or no reduction factor to maintain a high pixelcount, while image sensors located towards the edge of the array can usea large reduction factor to reduce the pixel count and thus provide ameans to increase the full-frame video recording rate.

In some embodiments, the data transfer process can be applied to systemswith multiple high volume data acquisition streams, such as to acomputational microscope system of a micro-camera array microscope(MCAM) system. Details about the MCAM system can be found in patentapplication Ser. No. 16/066,065, filed on Jun. 26, 2018; and in patentapplication Ser. No. 17/092,177, filed on Nov. 6, 2020, entitled“Methods to detect image features from variably-illuminated images”;hereby incorporated by reference in their entirety, and brieflydescribed below.

The MCAM system can be viewed as a group of multiple individualmicroscopes tiled together in an array to image a large sample. Theindividual microscopes can be configured into a micro camera package,e.g., forming a tightly packed array of micro-cameras with highresolution (1-10 μm) over a large area (hundreds of square centimeters).The images taken from the individual micro cameras, which includeoverlapped image patches of the sample, can be stitched together to formthe image of the sample.

The MCAM system can include a programmable illumination system, such asa large array of light sources, with individual light sources or groupsof light sources capable of being controlled separately, for example, bya controller. The light sources can be visible light sources, infraredlight sources or ultraviolet light sources such as light emitting diodes(LEDs) or lasers with appropriate wavelengths. The illumination systemcan be placed below or above the sample, to provide transmissive orreflective light to the micro cameras.

The MCAM system can use multiple micro-cameras to capture light frommultiple sample areas, with each micro camera capturing light from asample area sequentially from multiple patterned illuminationconfigurations provided on the same sample area.

The illumination system can provide the sample with differentillumination configurations, which can allow the micro cameras tocapture images of the sample with light incident upon the sample atdifferent angles and wavelengths. The illumination angle and wavelengthare important degrees of freedom that impacts specimen featureappearance. For example, by slightly changing the incident illuminationangle, a standard image can be converted from a bright field image intoa phase-contrast-type image or a dark field image, where the intensityrelationship between the specimen and background is completely reversed.

Further, by providing the sample with different illumination lightangles and wavelengths, both intensity and phase information of theimaged optical field can be recorded, which can allow the reconstructionof an image, for example, with more information or higher resolution.The MCAM system can offer size, weight, complexity, and cost advantageswith respect to standard microscopes. The MCAM system may not requireany moving parts, and its micro-cameras fit within a compact spacewithout requiring a rigid support structure and can thus operate withina small, confined space.

In some embodiments, the microscope system can include camera unitsconfigured to pre-process the captured image data, for example, toperform image reduction based on a preset factor or on an instructionfrom a user. For example, a camera unit can include an integrated moduleconfigured to preprocess data captured by the image sensor. Theintegrated module can be configured to reduce delays in data transfer.

In some embodiments, the pre-processing process can reduce the size ofthe captured images by skipping pixels, by binning pixel intensities, byskipping pixels with small intensity variations in space or time, bycompressing the image data, or by any combination thereof.

FIG. 8 illustrates a schematic microscope system according to someembodiments. The MCAM system 800 can include an array of camera units810 and a patterned illumination source 821 and 822 for microscopicimaging. The camera units 810 are configured to image partiallyoverlapping field of views (FOV) 811 of image areas of a sample 820. Thepatterned illumination source 821 and 822 can be configured to provideradiation, e.g., electromagnetic waves including visible light, infraredand ultraviolet light, on the sample 820 from a plurality of angles andspatial patterns, so that the spatial-angular distribution of radiationreaching the sample changes over time.

The illumination source can include a bottom set of radiation sourceunits 821, a top set of radiation source units 822, or both bottom andtop sets of radiation source units 821 and 822. The illumination sourcecan provide illumination patterns to the sample 820 of the MCAM system800, in which there is either a transmission illumination through thebottom set of radiation source units 821, or a reflection illuminationthrough the top set of radiation source units 822, disposed near themicro cameras. The illumination source can also provide a dualillumination geometry, in which there are a transmission illuminationthrough the bottom set of radiation source units 821, and a reflectionillumination through the top set of radiation source units 822.

The illumination source can be configured to generate one or moreillumination patterns. At each illumination pattern in thespatial-angular distribution of radiation generated from theillumination source 821 and 822, each camera unit can acquire an image.The set of images acquired from the camera units for the image areas forthe illumination patterns can be processed to form an imagereconstruction of the sample. The image reconstruction can also offer atleast a measure of sample depth, spectral (i.e., color) properties, orthe optical phase at the sample plane.

The MCAM system 800 can include a controller for controlling the cameraunits, the radiation source units, and for processing the images. Forexample, the controller can include a central processing units 880,which can couple to a camera and light controller units 884 forcontrolling the camera units, e.g., to tell the camera units when tocapture images, and for controlling the radiation source units, e.g., totell the radiation source units when to be activated and what radiationsource units to be activated. The central processing unit 880 can becoupled with the camera units to obtain the image data captured by thecamera units. The data can be stored in memory 881, can be processed ina post processing dataset 892, and can be displayed 883 on a display orto send to a final storage.

FIGS. 9A-9B illustrate a configuration for a microscope system accordingto some embodiments. FIG. 9A shows a perspective view and FIG. 9B showsa cross section view of a microscope system. The microscope system caninclude a camera array and an illumination source, which are controlledby one or more controllers, such as a camera controller, an illuminationcontroller, and a system controller.

An MCAM system can include an array of camera units 910 focused on alarge sample 920 under the illumination of an array of radiation sourceunits 921 and 922 such as LEDs or other light sources. The camera unitscan be configured to pre-process the captured image data, for example,to perform image reduction based on a preset factor or set of presetfactors or on an instruction from a user. A controller can be used tocontrol the illumination system to provide variably illuminatedradiation patterns, e.g., multiple illumination patterns with eachpattern different from other patterns, to the sample.

Each micro-camera images a unique FOV of the object of interest, forexample, a camera unit in the camera array can focus on a sample area,having overlapping areas 911 with nearby camera units to allow for imagestitching and image fusing. Each camera can acquire multiple imagesunder different illumination patterns. The captured images under oneillumination pattern can be combined together via an image stitchingalgorithm, for example, to form a composite image that has a larger FOVthat the FOV of just one of the micro-cameras individually. In someembodiments, the images from all of the k unique micro-cameras withinthe array may be stitched together to form a final composite image witha FOV that is approximately k times larger than the FOV of any of theindividual micro-cameras. In some embodiments, the captured images underdifferent illumination patterns can be fused together to form a highdimension image reconstruction of the sample.

The imaging process can start by capturing a set of images from multiplecamera units in the camera array under an illuminated pattern. Forexample, the illumination pattern can include radiation from one or moreradiation source units.

In some embodiments, to obtain high quality image reconstruction, theimaging process can capture k sets of variably-illuminated images from killuminated patterns. The variably illuminated radiation patternsinclude multiple illumination patterns in which the radiation is fromdifferent radiation source units. For example, an illumination patterncan include radiation from a single radiation source unit.

Thus, if the radiation source unit in an illumination source isactivated one at a time, there can be k illumination patterns for anillumination source having k radiation source units. For example, anillumination source can have 50 radiation source units. When theradiation source unit is activated one at a time, there are 50illumination patterns, with each pattern including radiation from aradiation source unit.

In some embodiments, different illumination patterns can be provided bya programmable array of radiation source units 921 and 922, withdifferent radiation source units activated to emit radiation to thesample. Each radiation source unit can be configured to cover the wholesample, e.g., radiation from a radiation source unit can reach all areasof the sample. The programmable radiation source array can includeradiation source units that are at different positions above or belowthe sample, as well as radiation source units that emit differentfrequencies (i.e., colors) of radiation. The radiation source units caninclude light emitting diodes (LEDs), individual lasers, laser diodes,spatial light modulators or other electronically controllable lightemission elements.

The variably-illuminated images can be captured by a camera array, witheach camera unit 910 in the camera array capturing an image. The cameraarray can include n camera units, with each camera unit configured tocapture an area of the sample. Adjacent camera units can be configuredto capture images having an overlapped area 911 images. The n cameraunits can be arranged to capture an image of the whole sample, having noverlapping images.

The camera units in the camera array are all activated at a same timefor each illumination pattern to capture images. Thus, for eachillumination pattern, there can be a set of n images. For example, acamera array can have 50 camera units. There is a set of 50 imagescaptured under one illumination pattern. The set of images can bestitched together to form an assembled image of the sample.

Under k different illumination patterns, there can be k sets ofassembled images, each captured under a different angle and spatialpattern. The image reconstruction of the sample, obtained by fusing thek sets of assembled images, can incorporate knowledge of thespatial-angular distribution of radiation reaching the sample. As such,the image reconstruction can offer a measure of sample depth, spectralproperties, and the optical phase at the sample plane.

FIG. 10 illustrates a flow chart for operating a microscope systemaccording to some embodiments. The microscope system can include acamera array and an illumination source, which are controlled by one ormore controllers, such as a camera controller, an illuminationcontroller, and a system controller configured to process the imagescaptured by the camera array under multiple illumination patternsgenerated by the illumination source. The camera units can be configuredto pre-process the captured image data, for example, to perform imagereduction based on a preset factor or on an instruction from a user. Theimage processing process can include a stitching process to stitchoverlapped image areas to form an assembled image of the whole sample.The image processing process can include a fusing process to fuse a setof assembled images under multiple illumination patterns to form animage reconstruction of the sample.

Operation 1000 generates an illumination pattern on a sample. Theillumination pattern can include radiation from one or more radiationsource units from the illumination source. The illumination pattern canreach and interact with the sample, such as being partially or totallyabsorbed by the sample, being phase-shifted, phase-delayed or otherwisedirectionally altered by the sample, being dispersed by the sample,being transmitted or reflected by the sample, depending on the locationof the light sources, being scattered by the sample, or a combination oftwo or more interactions, such as partially absorbed and partiallytransmitted.

Operation 1010 captures images from the camera units, for example, nimages can be taken for a camera array having n camera units. The imagescan be overlapped, meaning adjacent cameras can capture some image datafrom a same image area (a spatially overlapping area from within theimaging plane or volume). The images can also be non-overlapped andobserve different areas of the sample. Movement of the sample by asample stage can be used to bring the sample into view at differentmicro-cameras while multiple images are acquired by each micro-camera,such that the entire sample area may be observed within the capturedimage data for subsequent processing. The cameras can be positioned sothat the camera array covers the whole sample, e.g., all areas of thesample are captured by one or more camera units. The captured images canbe resized or reduced, for example, by integrated modules in the cameraunits.

Operation 1020 extracts and describes features to form featuredescriptors for the n images captured by the camera units under theillumination pattern. A feature can be described by a descriptor.

Operation 1030 stitches the images into an assembled image of the sampleby matching the descriptors of the extracted features. For example, thefeature descriptors in each image can be matched against featuredescriptors in other images, and the images can be transformed to allowthe overlapping of the matched descriptors to form an assembled image.After the stitching process, the assembled image can include nindividual images captured by the n camera units under the illuminationpattern.

A first step in the stitching process can include matching theidentified descriptors in each image of the n captured images. Thedescriptors have been extracted and described for the features on eachimage. The matching of descriptors can allow the alignment of the imagesto form an assembled image. After matching the descriptors, the imagescan be aligned to form the assembled image.

The identification and matching of descriptors can be performed onmultiple images, or alternatively, on individual images. For multipleimages, two images can be stitched together, even though there are nomatched features on the two images. The two images can be stitchedtogether based on the matching of the descriptors, which can perform thealignment for all images.

An assembled image of the sample can be an image representing an area ofinterest on the sample. In some embodiments, the sample image can beconsidered as the image of the whole sample, since the whole sample istypically imaged. In some embodiments, only an area of interest on thesample is imaged.

The sample image can be large, e.g., larger than an image captured by asingle camera of the camera array. Further, the sample image isconfigured, e.g., positioned in the microscope, in such as way so thatthe images captured by the camera array cover the area of interest onthe sample, e.g., stitching the images captured by the camera array canallow the formation of a sample image.

Operation 1040 optionally repeats for a total of k differentillumination patterns. The repetition process can include generating adifferent illumination pattern, taking n more images from the n cameraunits, optionally performing image reduction on one or more of theimages to reduce the image size as desired, extracting features andstitching the images together to form another assembled image. Therepetition process can continue until there are a total of killumination patterns, e.g., when the number of illumination patternsreaches a predetermined number. The repetition process can also continueto capture video of dynamic samples that change or move as a function oftime.

Each illumination pattern can be different and the set of illuminationpatterns can be configured to provide an increase in the amount ofsample information to be captured by the cameras. After k illuminationpatterns, there can be a set of k assembled images, with each assembledimage including n images captured by n camera units under anillumination pattern, stitched together.

Operation 1050 partitions the set of k assembled images into patches.The partition process can partition a sample image into multiplepatches. Thus, the partition process can partition the set of assembledimages into multiple sets of patches, with each patch in a set ofpatches being the same partitioned area in each sample image of the setof assembled images. The patch partition can be selected to simplify thereconstruction of fused images from the sets of patches.

Each set of patches can include multiple patches, with a patch being thearea portion on a sample image of the set of sample images. A patch canbe a portion of a sample image, which can be smaller than the imagecaptured by a camera. Thus, an image captured by a camera can includeone or more patches. A patch can be at an overlapped area betweenmultiple cameras, and thus there can be multiple patches representing anarea on the sample image that is captured by multiple cameras. Forexample, if the patch is at the overlapped area of 2 cameras, there aretwo patches in a sample images stitched together from all cameras for aparticular illumination pattern. For k illumination patterns, a set ofpatches can include 2 k patches, for areas of the sample overlapped bytwo cameras.

Operation 1060 fuses the multiple patches in each set of patches intofused patches. For example, the patches in a set of patches can be fusedtogether to form a fused image of the patch. The fusing process thus canproduce a high resolution image for the patches.

Each set of k images from each camera can be fused, e.g., processed tocombine into one image. Since each image of the set contains differentinformation about the image area, such as the transmissive or reflectivelight data and the scattered light data from different light angles,images in the set can be processed to combine this information to afused image carrying both intensity and phase information.

In the fusing process, a complex function S, e.g., having intensity andphase information, representing the sample area, can be determined fromthe set of k images. When an image of the sample area is captured by acamera, only the intensity is captured, e.g., the phase information islost to the camera, and a 2D image is formed to represent the 3D samplewith finite thickness. By taking a set of images of a sample underdifferent illumination patterns representing light having uniquespatio-angular distributions, each with an optical phase that can varyas a function of space, the phase information of the sample can bereconstructed to generate a representation of the 3D sample with finitethickness.

For example, the sample area represented by the set of k images can bedescribed as a complex function S. The MCAM system can capture k imagesof the sample area, meaning the complex function S is transformed into aset of k images M through a system matrix T that describes the MCAMimage formation process, which can determined from the geometry of theMCAM setup, including the light paths from the light sources to thecamera.M=∥T·S∥ ² +n

The set of k images M is the result of the transformation of the systemmatrix M with the sample function S. Here, the absolute square term isdue to the ability to detect only intensity by the camera, and n is anadditive Gaussian noise.

The sample function S can be calculated from the above function bysolving the inverse problem. A possible approach for the inverse problemis to minimize the mean-squared error between the measured magnitudesand an estimate of the sample function. Another possible approach is tominimize a related negative log-likelihood function, which is based on aPoisson noise prior. Another possible approach is to treat the problemas a cost function, using appropriate minimization algorithms, such as aDouglas-Rachford algorithm.

In some embodiments, an approach for the inverse problem is to solve theminimization problem by constructing an Augmented Lagrangian and thenminimizing the Augmented Lagrangian with gradient descent. In someembodiments, the inverse problem can be solved using an iterativeoptimization strategy that first determines the gradients, or thegradients and the Hessians, and then applying a Gauss-Newton method. Insome embodiments, the sample may be fluorescent and function S can be areal, positive-valued function, and a minimization method similar tothose used in structured illumination fluorescent microscopes todetermine a high-resolution sample can be used. In some embodiments, anapproach for the inverse problem is to use a neural network, such as aconvolutional neural network or a deep neural network, either trained oruntrained, to transform the measured data into The image reconstructionsprocess can also include an estimate of the sample height, the samplephase, and its spectral color content.

The fused patches can then be assembled to produce a final, large,high-resolution image reconstruction of the sample. The final image caninclude a measurement of depth at different spatial locations across theimage. The final image can include a measurement of the optical phase atdifferent spatial locations across the image. The final image caninclude a measurement of multi-spectral content at different spatiallocations across the image.

FIG. 11 illustrates an operation schematic of a microscope systemaccording to some embodiments. A microscope system 1100 can haveillumination sources 1121 and camera units 1110 for capturing images ofportions of a sample. As shown, the illumination source 1121 can includek=3 radiation source units, and the camera units 1110 can include n=3camera units.

In some embodiments, each radiation source unit is configured to coverthe whole sample. Each camera unit is configured to capture images of aportion of the sample. Further, a camera unit captures only imagescorresponding to a matching radiation source unit. For example,radiation source unit 1121 can irradiate the whole sample. All cameraunits, such as camera unit 1110, can capture images A, B, and C ofoverlapped portions of the sample. Thus, the images A, B, and C capturedby n=3 camera units 1110 can be assembled, such as stitching together,to form an assembled image 1126, e.g., a complete image of the sample,under the illumination of the radiation source unit 1121. Thus, underone illumination pattern generated by a radiation source unit 1121, acomplete image of the sample can be captured. The process can berepeated for the k=3 radiation source units to form a set of images1127.

In some embodiments, the present invention discloses systems and methodsof imaging for high frames and high frame rates by using an array ofmultiple small microscopes (i.e., micro-cameras), tiled together in anarray. Using preprocessing modules, such as integrated with imagesensors in the micro-cameras, an image reduction process can beperformed, resulting in high fame rates without affecting the datatransfer process. Using a tightly packed array of micro-cameras havinghigh resolution (1-10 μm) over a large area (up to hundreds of squarecentimeters), large frames with high pixel counts can be achieved. Byconfiguring the camera array to transmit image data in multiple paralleldata streams, the data transfer rate can be minimally affected.

The radiation from different illumination patterns can provide radiationwith different path lengths to the sample, so that each point on thesample can receive radiation with multiple path lengths. The differentpath lengths can allow the reconstruction of phase information usingintensity information from the multiple path length radiation. Thedifferent illumination patterns can enable image reconstruction of thesample, e.g., images with higher quality such as having depthmeasurements or higher resolution. The system can be versatile,providing large image frames, high frame rates, or high image quality,for example, through an optimization of number of cameras, imagereduction process or factors, or number of illumination patterns.

The camera array can include multiple camera units disposed above thesample. A camera unit can include a camera, e.g., an optical sensorcapable of capturing images due to radiation in a wavelength range, suchas images caused by visible light, infrared light, or ultraviolet light.The camera unit can include a pre-processing module, which is configuredto perform image reduction when reading pixel data from the imagesensor. The camera unit can be configured to receive instructions from auser, such as through a central processing station, with theinstructions including image parameters and image reduction factors, forconfiguring the camera unit.

The camera units are positioned so that each camera unit is configuredto capture images of an area of the sample, e.g., different camera unitsare configured to capture images of different areas. The areas capturedby different camera units can be partially overlapped, e.g., adjacentcamera units can have an overlapped field of view. The overlapped fieldof view can be greater than 50% for image reconstruction (e.g., phasecalculation for the captured images), or can be less than 50% for imagestitching (e.g., matching features for assembling a larger image fromsmaller images). The multiple camera units with overlapped field of viewcan allow the microscope system to capture images of large sample, bystitching and assembling images of portions of the sample.

The controller can be configured to control the camera units to captureimages from the sample under each illumination pattern from theillumination source. By capturing images from multiple differentillumination patterns, phase information can be reconstructed from themultiple captures intensity-only images. With a large overlapped fieldof view, such as greater than 50%, the image reconstruction process canbe reliable and can provide an image reconstruction with additionalinformation of the sample such as depth information.

After multiple illumination patterns, the system can have multipleassembled images, with each assembled image including multiple imagescaptured by the camera units under one illumination pattern and stitchedtogether. The assembled images can be fused together, e.g., havingunderwent an image reconstruction process, to form an imagereconstruction of the sample, which carries more information than theindividual assembled images, such as having phase information inaddition to intensity information. Thus, the controller can beconfigured to use a fusing algorithm based on overlapped field of viewsof two adjacent camera units.

In some embodiments, the camera units can be arranged in one or moreperiodic arrays, for example, separated by a gap between the arrays toaccommodate the radiation source units. The individual camera units canbe disposed uniformly or evenly in the periodic arrays.

FIGS. 12A-12C illustrate a sequence of data flow from a camera arrayaccording to some embodiments. The sequence represents the multiplecapabilities of a system, such as a microscope system, having a cameraarray 1210X, 1210Y, and 1210Z with image reduction ability, and anillumination source having radiation sources 1221A, 1221B, and 1221C.

In FIG. 12A, radiation source unit 1221A can shine light on a sample1220. The radiation source unit 1221A is configured to cover the wholesample, or at least the portion of interest in the sample. Under theillumination pattern generated by the radiation source unit 1221A, eachcamera 1210X, 1210Y, and 1210Z of the camera array can capture an image,such as images 1250AX, 1250AY, and 1250AZ, respectively. The capturedimages can be pre-processed, under an image setting of the cameras, forexample, to perform image reduction, using a reduction process such asfiltering, sub sampling (e.g., skipping pixels), binning (e.g.,averaging pixel intensities in bins), gradient cutoff (e.g., removingpixels having low spatial or temporal gradient), or compression. Theresized images 1263AX, 1263AY, and 1263AZ, preprocessed from thecaptured images 1250AX, 1250AY, and 1250AZ, respectively, can be sent inmultiple parallel data streams to a process module or to a centralprocessing station.

In FIG. 12B, the process can be repeated for another illuminationpattern, such as an illumination pattern generated by radiation sourceunit 1221B. The cameras 1210X, 1210Y, and 1210Z can capture images underthe new illumination pattern, such as image 1250BX captured by camera1210X. The captured images can be preprocessed to form reduced images,such as image 1263BX reduced from captured image 1250BX.

In FIG. 12C, the process can be repeated for another illuminationpattern, such as an illumination pattern generated by radiation sourceunit 1221C. The cameras 1210X, 1210Y, and 1210Z can capture images underthe new illumination pattern, such as image 1250CX captured by camera1210X. The captured images can be preprocessed to form reduced images,such as image 1263CX reduced from captured image 1250CX.

The multiple illumination patterns can form multiple parallel datastreams 1235 from multiple cameras, with each data stream having asequence of reduced images captured under the series of illuminationpatterns.

In some embodiments, the system can be configured to vary the number ofillumination patterns and the image reduction parameters to obtain adesired image quality and a desired frame rate. For high imagequalities, the number of illumination patterns can be high. For highframe rates, the image reduction factor can be high, such as having alow sub sampling value. For example, using 1 illumination pattern, astandard quality image can be obtained, with a frame rate of 6 framesper second to create a video with 6 frames per second. Using 24illumination patterns, a high quality image reconstruction can beobtained, providing depth measurements and resolutions exceeding thediffraction limit. However, the frame rate can be low, e.g., reduced bya factor of 24, for 4 seconds per effective frame. Note that the 24illumination patterns can be shown in a repeated manner to producesubsequent effective frames at a rate of 4 seconds per effective frame.Using an image reduction method such as sub sampling at a rate of 1/16,e.g., one pixel saved for 15 pixels skipped in a 4×4 sub samplingpattern, the frame rate can increase by a factor of 16, for 4 effectiveframes per second frame rate, in the case of high quality imagereconstruction.

FIG. 13 illustrates a microscope system configured with an imagereduction capability according to some embodiments. A microscope system1300 can include a camera array 1310, which includes multiple cameraunits. A clock generator 1314 can be used to generate a common clocksignal, such as a 25 MHz clock signal, to all camera units, for example,through clock buffers. The camera units can have preprocessed modules1313, which can be configured to preprocess the image data when readingfrom the image sensors of the camera units. The preprocess modules canperform an image reduction process, such as sub sampling with optionalfiltering, binning with optional filtering, low intensity pixel gradientremoval (e.g., calculating intensity spatial or temporal gradient forthe pixels, such as through a derivative calculation or one or moreconvolution operations with different convolution filters, and removingpixels having resulting values below a threshold value), datacompression, or any combination thereof. The image reduction process canbe triggered with specified processes and reduction factors throughinputs 1342 from a user, such as through a process module 1320 orthrough a computational unit. The reduced images can be sent, inmultiple parallel data streams 1335, to a process module 1320, which isconfigured to organize the image data. The multiple parallel datastreams can significantly reduce the transfer time of the captured andreduced images from the camera units to the process module 1320. Forexample, through a common clock with similar image parameters, themultiple data streams can have minimum variation in timing, which canallow the process module to receive multiple data streams in the sametime as receiving one data stream.

A process module 1320, such as an FPGA based module (e.g., a modulecontaining a processing chipset, such as an FPGA, or other chipset of anASIC, an ASSP, or a SOC), can be configured to receive image data fromthe multiple camera units, e.g., through data streams 1335, which use ahigh speed serial communication link for each camera unit. The FPGAbased module 1320 can include a shallow buffer 1323, for example, tostore incoming data from the data streams 1311. The FPGA based modulecan be configured to send 1342 sensor configuration data to the cameraarray, for example, to provide image parameters to the image sensors ofthe camera units. The sensor configuration can be received from acomputational unit having a processor 1330 and a memory 1331, forexample, through communication 1341. For example, the processor can sendconfiguration and settings to the FPGA based module, with theconfiguration and settings including setting information for the FPGAbased module and the configurations for the image sensors. The FPGAbased module can communicate 1345 with the computational unit usingdirect memory access (DMA) to pass data directly to the memory 1331,through a high speed link such as PCIe. The FPGA based module cancommunicate 1324 with a module 1332, which can be configured to controllighting, motion, and sample handling for the microscope system. Thecomputational unit 1330 can also communicate directly 1333 to the module1332. The computational unit 1330 can communicate with a storage ornetwork devices (not shown). The system can include peripheral devices,such as stages, illumination units, or other equipment involved in theapparatus necessary to ensure adequate imaging conditions.

The microscope system 1300 can be configured to minimize the delay inanalyzing the data captured by a multi-sensor array 1310 in themicroscope, such as from the time an operator defines the microscope'sdynamic settings to the time the acquired images are analyzed. Themicroscope is configured to capture multiple images simultaneously frommultiple image sensors, and thus can generate gigabytes of image dataper second, with each snapshot containing a few hundred megabytes ofdata. With minimum delay time in the data transfer process, acomputational unit having a processor in the microscope system canobtain high frame rates for the captured images, which can allow anobservation of fast moving or fast reacting organisms.

In some embodiments, the microscope system with minimum data transferdelay can be constructed through a combined design of electronichardware, firmware and analysis software on the processor of themicroscope system. For example, multiple image sensors can be placed ona single printed circuit board and share a common clock 1314 that isused to provide the multiple image sensors with common clock buffers, acommon frame rate and imaging parameters. The data from the imagesensors can be send in multiple parallel data streams to a processmodule 1320, such as an FPGA, ASIC, ASSP, or SOC based module for dataaggregation and preparation before sending to a memory 1331 associatedwith the processor 1330 of the microscope system for data analysis. TheFPGA based module can also be configured to perform direct writing 1345to the memory without using processing time from the processor, whichcan ensure that the image data acquired from the multiple image sensorsis not lost while the CPU is busy analyzing data and cannot receive datafrom the sensors. Furthermore, the FPGA based module can enable theprocessor to be connected to many image sensors without the need for acustom processor design, since processor I/O are often designed forspecific very high speed communication standards, while image sensorsmay require completely different communication protocols.

In some embodiments, the data flow is configured in adequate packetsizes, which are designed to be efficiently transferred to acomputational unit. Further, for the stored data to be efficientlyutilized by the processor, such as a central processing unit (CPU), orother computational unit such as a graphical processing unit (GPU), datafor each image must be stored in contiguous memory on the memoryassociated with the processor.

In some embodiments, the multiple data streams from the camera units areorganized in a serial data packet stream to send to the memory of thecomputational unit, for example, by an FPGA based module. It is aninnovative feature of the data flow architecture that the incoming datastream to the memory of the computational unit contains a sequence ofdata packets in the size of a partial image frame, e.g., a small portionof a full or reduced image frame data, such as a data packet size thatis 0.01% to 1%, 0.05% to 0.5%, 0.02% to 0.4%, or 0.1% to 0.2% of a fullor reduced image. For example, an image sensor can have 4096×3100pixels, with each pixel measuring one byte of data per frame. A datapacket size can be chosen to include one line of the pixel size, e.g.,4096 bytes, for 1/3100 times smaller than a full image data. A datapacket size can be chosen to include two lines of the pixel size, e.g.,2×4096 bytes, for 1/1550 times smaller than a full image data. A datapacket size can be chosen to include four lines of the pixel size, e.g.,4×4096 bytes, for 1/775 times smaller than a full image data. Forreduced images, such as image reduced by a factor of 16, the data packetsize can still be selected to be 4096 bytes, which translates into 4lines of a reduced image size, such that the reduced image size has1024×775 pixels.

In the small data packet architecture, e.g., the size of a data packetis less than 0.5% of a full image size, the FPGA based module configuredto reorganize the multiple incoming data streams from the multiplecamera units into a data packet stream can only need to store one or twodata packets from each data stream of the camera unit, such as a fewmultiples of 4096 bytes of data, for image sensors having a 4096×3100pixel size.

The small data packets can be critical in the flow of data from multipleparallel data streams from multiple camera units to the memory of thecomputational unit. The FPGA based module can configured and organizethe image data into small partial image frames, e.g., data packets, andtransfer the partial image frames to the memory so that the partialimage frames are available to the processor. With the processorconfigured to process and analyze the partial image frames, instead ofwaiting for full image frame data, the responses from the processor canbe thousands of times faster, since the partial image frames can bethousands of times smaller, and thus can arrive thousands of timesfaster.

An advantage of the data flow architecture using small data packets isthat image data, actually data for partial image frames, can beavailable for the processor to analyze with minimum delays after theimages are acquired by the camera units. With the partial image framesthousands of times smaller than full image frames, the image flow fromthe image sensors to the processor can also be thousands of timesfaster. The fast data flow can enable the processor to start analyzingthe image data, e.g., the first portion of the images, before the fullimages have completed the transfer process, and thus can allow theprocessor to change the image parameters of the image sensors before theacquisition of next images.

Also the ability to start analyzing images, e.g., partial image frames,immediately as the image being captured and data coming out of the imagesensors can avoid the dead time between image acquisition, processingand imaging parameter selection. For example, changing the illumination,sample position, or other environmental parameters can often be a slowprocess and ensuring that the new imaging parameters are quicklyidentified is an important factor to reducing the total imaging time ofan experiment.

In addition, the fast data flow which allow the availability of imagedata, e.g., partial image frames, with minimal delays after the images,e.g., full image frames, being captured can be critical for manyapplications in microscopy where decisions must be made depending on theobserved measurements from a sample, such as in biological applicationswith stimuli applied to a model organism to understand the behavior ofthe organism under different scenarios. The fast data flow process canallow the processor to quickly change 1325 the observed parameters,e.g., the imaging parameters of the sensors or accompanying hardware,such as the illumination, sample stage, sample temperature, etc., tobest observe the organism behavior.

In some embodiments, after an image acquisition is triggered 1342, withinformation relating to image reduction, either by continuousacquisition, or an event of interest, data from each image sensor beginsto arrive continuously in time, from all image sensors at the same time.To avoid lengthy delays, the receiving module, e.g., the FPGA basedmodule, can create small data packets that can be efficientlytransmitted to the memory. Various different packet sizes are suitable,but a packet size of 4096 bytes (0×1000 in hexadecimal representation)is particularly interesting since it aligns well with a page of memoryon the computational unit. For image sensors with a row size of 4096pixels, when the image is transmitted with 8 bits per pixel, this onepage of memory on a computer corresponds to one row of data from animage sensor. For image sensors with different sizes, or for reducedimage sizes, 4096 bytes can correspond to partial rows of data from theimage sensor, or multiple rows of a sensor. A key design parameter isthat the FPGA can gather multiple such packets simultaneously from thedifferent image sensors. After receiving enough data in each of thedifferent image sensors, the data packets can be reordered in time, in around-robin fashion, so that they are transmitted one after the other toa high speed communication output link 1345 to the memory. Data packetsin the high speed communication link can be sent to the appropriatelocation in the computer memory where it can then be analyzed, stored,or forwarded to another computer, by the end user application.

In some embodiments, the data flow of a sequence of data packets eachcontaining a partial image frame can be formed by an FPGA based modulehaving a small data buffer 1323. For example, the FPGA based module canstore a small partial image frame, such as multiples of a line of data nan image sensor, from each image sensor. The partial image frame dataflow can lower the cost of the FPGA based module, by requiring smallbuffer size FPGA based module. For example, in a system having 54 imagesensors of 4096×3100 pixel size, with each pixel stored in a byte ofdata, full image frames for the 54 image sensors can require almost 700Mbytes of data for a buffer in a FPGA based module. Using the partialimage frame data flow, with each partial image frame having 4 lines ofimage data, the partial image frames for 54 image sensors can requireless than 1 Mbytes, which can be easily achievable, as compared to the700 Mbytes requirement. Further, 700 Mbytes of buffer space requiresdedicated hardware to ensure this large buffer space, increasing thesize, cost, and complexity of the hardware design.

In some embodiments, the data flow can accommodate a large number ofimage sensors, e.g., a large number of parallel data streams from acamera array, based on the small size of the partial images. Asdiscussed above, a one-Mbyte FPGA based module can accommodate 54 imagesensors. FPGA based modules with higher buffer sizes, such as multiplesof 1 Mbyte sizes, can accommodate multiples of 54 image sensors.

In some embodiments, the FPGA based module is configured to receivedata, e.g., full images or reduced images depending on image reductioninstructions, from multiple data streams from multiple camera units, andthen reorganize the received data on the fly, to generate an outputstream of data packets to be sent to appropriate locations in the memoryassociated with the processor. To minimizing delays, the FPGA basedmodule can send the data packets directly to the memory locations,bypassing the need for any involvement of the processor. The directmemory writing can be accomplished by a direct memory access feature onthe computational unit, with the FPGA based module using a communicationlink that support direct memory access (DMA) together with having anappropriate DMA engine capable of writing the data packets to theappropriate locations on the memory.

Further, the FPGA based module also needs to know the memory locationsto send the data packets. Thus, during the system set up phase, theprocessor can send 1340 descriptors of the locations of the memory,e.g., the RAM associated with the processor, to the FPGA based module.The processor can send 1324 set up information to a configuration andsetting module 1332, with the FPGA based module responsible for triggerand synchronization 1342. For example, once the imaging parameters aredefined, such as frame rate, image size, triggering conditions, numberof cameras used, exposure time, digital gain, analog gain, number offrames to be acquired, and image reduction information, the processorcan provide the memory addresses where each image must be stored. Thelocation of these addresses is designed such that each image will belocated contiguously in the memory to ensure efficient computation andsubsequent analysis. The FPGA based module and the computational unitcan be configured to allow the FPGA based module to send data packets tothe memory of the computational unit without requiring any additionalmemory copies from the computational unit.

In some embodiments, the location descriptors, such as the startingaddresses or links to the starting addresses, of the memory can beincorporated in the data packets, so that each data packet can be sentto the appropriate memory locations. The FPGA based module cancommunicate with the computational unit through an interface thatsupports direct memory access, which can require hardware support fromthe computational unit, and can require a link that support directmemory access, such as PCIe or similar interfaces. The direct memoryaccess feature can enable the FPGA based module to control thedestination of the large quantities of data packets that are transmittedto the memory of the computational unit.

After organize the data packets from the multiple parallel incoming datastreams from the camera array, the FPGA based module can communicatedirectly with the memory, freeing the processor for other tasks such asprocessing or analyzing image data. Direct memory access can avoidmultiple copies of the data formed between the operating system kernel,which can limit control to the external hardware and the end userapplication that lives in an isolated environment outside of kernelmemory. The direct memory access can prevent loss of data, due to anefficient and predictable memory writing process. With direct memoryaccess, the FPGA based module can ensure that data packets containingpartial image frames of the images captured by the camera units canarrive at their respective locations in the memory. For optimalcomputation speed, the memory can reserve a series of addresses thatcontain the starting locations of the N*M buffers (where N is the numberof image sensors and M is the number of images to capture per sensor),each large enough to store a single image. In addition, for efficientdirect memory access, the data packet size is optimized to the operatingsystem boundaries, such as integer multiples of 4096 bytes.

In some embodiments, the FPGA based module can be configured to havematch bandwidths between the outgoing data packet stream to the memoryand the multiple incoming streams from the camera units. For example,the data output interface, which is the communication link between theFPGA based module and computational unit, can match with or have greaterthan, in terms of aggregate data throughput, the data input interfaces,which are the communication links between the FPGA based module andmultiple camera units. The communication links and the number of lanesfor the communication links can be selected to make sure that there isno data bottlenecking in transferring data packets to the memory.

Benefits of the data flow architecture and the microscope systemsincorporating the data flow architecture, can include a new ability torapidly record high-resolution microscopy imagery over a very largefield of view, in addition to high image frame rates, using a multitudeof micro-cameras while minimizing the memory requirements on the datagathering FPGA based module. This new capability opens up new directionsfor scientific discovery—allowing for image snapshots or videos of largearea that may contain a variety of samples including but not exclusiveto freely moving organisms over more than just several squarecentimeters while simultaneously changing their stimuli or imagingenvironment. It also provides a more efficient manner to obtain largedata volumes from large, flat objects (e.g., semiconductor wafers,sheets of glass, plastic components) while their imaging environment isadapted to the recorded images or partial images.

FIGS. 14A-14B illustrate flow charts for forming microscope systemsaccording to some embodiments. In FIG. 14A, operation 1400 forms acomputational multi-camera system, with the system including a processmodule configured to organize data received, in parallel, from multiplecameras of a camera array into a sequence of data packets to send tomemory of the system. Each camera can be configured to preprocesscaptured images to reduce sizes of the images. Each data packet caninclude a portion of a reduced image. Bandwidth of the data packetsequence to the memory can be greater than or equal to combinedbandwidths of the multiple parallel data from the multiple cameras. Theprocess module can be configured to send the data packets directly tothe memory without using a processor of the computational system.

In FIG. 14B, operation 1420 captures images by cameras of a cameraarray. Operation 1430 processes the images to reduce sizes of theimages, with the size reduction including at least one of a sub samplingprocess, a binning process, a low gradient removal process, or a datacompression process. Operation 1440 sends the processed images, inmultiple parallel data streams, to a process module for organizing intoa data packets. Operation 1450 sequentially sends the data packets to amemory of a central processing station using direct memory access, withbandwidth of the data packet stream matching combined bandwidths of themultiple parallel data streams.

Correlation with Image Frame Rates

In some embodiments, the system, such as a microscope system, can offerlarge frames (through multiple camera units in a camera array), togetherwith image reduction ability for high frame rates (through a preprocessmodules incorporated with the camera units) and variably illuminationpatterns for high quality images (through an illumination sourceoffering different illumination patterns). In addition, the system canoffer image optimization based on a selection of image reductionparameters and numbers of illumination patterns.

FIG. 15 illustrates a correlation between image reduction and imageframe rates according to some embodiments. For no image reduction, e.g.,the ratio of the reduce image size and the full image size is 1, a lowframe rate can be obtained at 1580A. With more image reduction, theimage is reduced more and more for a lower ratio of the reduce imagesize and the full image size, higher frame rates can be achieved, suchas at 1581A and 1582A.

The correlation between image reduction and image frame rates can allowa determination of image reduction parameters based on a desired imageframe rate. For example, for a full image capture process, a frame rateof 6 frames per second can be achieved. For a desired image frame rate,such as 24 frames per second, a reduction ratio of ¼ can be selected,e.g., one pixel can be saved for every 3 discarded pixels, such as in a2×2 pattern. Further, a reduction process can be selected, such as a subsampling process, a binning process, a process for removing low spatialor temporal gradient pixels, or a compression process.

In some embodiments, the system, such as a microscope system, can obtainhigh quality image through multiple illumination patterns. Further, thesystem can offer tradeoff between image quality and frame rate.

FIG. 16 illustrates a correlation between image reduction and imagequality with image frame rates according to some embodiments. To improveimage quality with fixed reduction parameters, the frame rates can bereduced. To improve image quality with a desired frame rate, thereduction parameters can be changed to suite the desired quality andframe rate.

For example, for no image reduction, higher image quality results inlower frame rates, as shown at 1680A to 1680B to 1680C. With a fixedframe rate, higher image quality can be achieved through a reduction inresized ratios, e.g., the reduced images are smaller and smallercompared to the full image, as shown in 1680A to 1681B to 1682C.

FIGS. 17A-17B illustrate flow charts for optimizing image parametersaccording to some embodiments. In FIG. 17A, image reduction parameterscan be determined for a desired image frame rate.

Operation 1700 receives a desired frame rate. Operation 1710 determinesat least one of an image reduction factor or an image reductionalgorithm based on the desired frame rate. Operation 1720 instructscameras of a camera array in a system to perform preprocessing oncaptured images with the image reduction factor.

In FIG. 17B, image reduction parameters and number of illuminationpatterns can be determined for a desired image frame rate. Operation1740 receives a desired frame rate and a desired image quality.Operation 1750 optimizes between an image reconstruction process and animage reduction process based on the desired frame rate and the desiredimage quality. Operation 1760 instructs cameras of a camera array in asystem to perform preprocessing on captured images with the optimizedimage reduction process. Operation 1770 instructs light sources of anillumination source in the system to generate a number of illuminationpatterns satisfied the optimized image reconstruction process.

While this invention has been particularly shown and described withreferences to example embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the inventionencompassed by the appended claims.

What is claimed is:
 1. A computational microscope comprising a cameraarray, wherein the camera array comprises multiple camera units, whereineach camera unit of the camera array is configured to capture images ofan area of a sample, wherein different camera units are configured tocapture images of different areas of the sample, wherein at least twodifferent areas having images captured by two different camera units arepartially overlapped, wherein each camera unit comprises a preprocessmodule configured to perform size reduction of the images captured bythe camera unit; an illumination source, wherein the illumination sourcecomprises one or more radiation source units; a controller, wherein thecontroller is configured to control the one or more radiation sourceunits of the illumination source to generate one or more illuminationpatterns, wherein the controller is configured to control the cameraarray to capture images of areas of the sample under an illuminationpattern of the one or more illumination patterns, wherein the controlleris configured to control the camera array to perform image sizereduction of the captured images, wherein the controller is configuredto control the camera array to send the captured images with or withoutsize reduction, wherein the controller is configured to form one or moresets of images received from the camera array, wherein each set ofimages comprises with-or-without-size-reduction images captured under anillumination pattern.
 2. A microscope as in claim 1, wherein thepreprocess module is integrated with an image sensor of each cameraunit.
 3. A microscope as in claim 1, wherein the size reductioncomprises an image sub sampling process, an image binning process, aremoval of low spatial or temporal gradient pixels, an image compressionprocess, or any combination thereof.
 4. A microscope as in claim 1,wherein the controller is configured to fuse the one or more sets ofimages into an image reconstruction of the sample.
 5. A microscope as inclaim 1, wherein the image size reduction is configured to meet an inputframe rate.
 6. A microscope as in claim 1, wherein the one or moreillumination patterns are configured to meet an input comprising aquality of image reconstruction.
 7. A microscope as in claim 1, whereina number of the one or more illumination patterns and the image sizereduction are optimized to meet a quality of image reconstruction whilesatisfying a desired frame rate.
 8. A microscope as in claim 1, whereinthe captured images with or without size reduction are configured to besent directly to a memory associated with the controller.
 9. Amicroscope as in claim 1, wherein the camera array is configured to sentthe captured images with or without size reduction to an process module,wherein the process module is configured to organize the images intodata packets to send to a memory associated with the controller, whereinthe process module comprises matched communication interfaces betweendata flow from the camera array and data flow to the memory.
 10. Amethod for operating a computational microscope, the method comprisingcapturing images of a sample by a camera array under one or moreillumination patterns generated by an illumination source, wherein thecamera array comprises multiple camera units, wherein each camera unitof the camera array is configured to capture images of an area of asample, wherein different camera units are configured to capture imagesof different areas of the sample, wherein at least two different areashaving images captured by two different camera units are partiallyoverlapped, wherein the captured images are preprocessed for sizereduction in the camera array based on an image reduction parameter;wherein the one or more illumination patterns are generated based on animage quality parameter; assembling the preprocessed images capturedunder the one or more illumination patterns received from the cameraarray, wherein a controller is configured to control the camera array tosend the captured images with or without size reduction, wherein eachset of images comprises with-or-without-size-reduction images capturedunder an illumination pattern.
 11. A method as in claim 10, wherein theimage reduction parameter comprises at least one of a sub samplingprocess, a binning process, a removal of pixels having a spatial ortemporal intensity gradient less than a threshold value, or an imagecompression process.
 12. A method as in claim 10, further comprisingfusing the preprocessed images into an image reconstruction of thesample.
 13. A method as in claim 10, further comprising inputting theimage reduction parameter and the image quality parameter.
 14. A methodas in claim 10, further comprising inputting a desired frame rate and adesired image quality, determining the image reduction parameter basedon the desired frame rate, determining the one or more illuminationpatterns based on the desired image quality.
 15. A method as in claim10, further comprising sending the preprocessed images in multipleparallel data streams to a process module to be organized into datapackets to be sent directly to a memory of a processor, whereinbandwidth of the data packets sent to the memory is greater than orequal to combined bandwidths of the multiple parallel data streams. 16.A method for operating a computational microscope, the method comprisingproviding an instruction to a camera array, wherein the instructioncomprises a desired image frame rate and a desired image quality,determining an image reduction parameter and a number of illuminationpatterns to generate images meeting the desired image frame rate and thedesired image quality, capturing images of a sample by the camera arrayunder the number of illumination patterns generated by an illuminationsource, wherein the camera array comprises multiple camera units,wherein each camera unit of the camera array is configured to captureimages of an area of a sample, wherein different camera units areconfigured to capture images of different areas of the sample, whereinat least two different areas having images captured by two differentcamera units are partially overlapped, wherein the captured images arepreprocessed for size reduction in the camera array using the imagereduction parameter; assembling the preprocessed images captured underthe number of illumination patterns.
 17. A method as in claim 16,wherein the image reduction parameter comprises at least one of a subsampling process, a binning process, a removal of pixels having aspatial or temporal intensity gradient less than a threshold value, oran image compression process.
 18. A method as in claim 16, furthercomprising fusing the preprocessed images captured under the number ofillumination patterns into an image reconstruction of the sample.
 19. Amethod as in claim 16, wherein the determination of the image reductionparameter and the number of illumination patterns is based on anoptimization between the desired image frame rate and the desired imagequality.
 20. A method as in claim 16, further comprising sending thepreprocessed images in multiple parallel data streams to a processmodule to be organized into data packets to be sent directly to a memoryof a processor, wherein bandwidth of the data packets sent to the memoryis greater than or equal to combined bandwidths of the multiple paralleldata streams.